With the explosion of Big Data, complex search queries can be slow when running against SQL database. The performance issue roots from the fact that very simple wildcard-based text search required full table scans which results in degradation of the site's overall performance.
Apache Lucene is a free open source information retrieval software library. Along with the Apache Solr which is the open source enterprise search platform were introduced to address this issue. Apache Lucene/Solr are made for any application which requires full text indexing and searching capability. They are widely recognized for its utility in implementation of Internet search engine and local, single-site searching. Solr is able to achieve fast search responses by searching on indexes rather than on texts.
Apache Solr core manages a single index. An index is a set of all data used to store information about document to be searched. Only one core is loaded at a time. A single Solr instance is capable of managing multiple indexes hence the name Multi-Core.
By default, users that have access to one of the cores may also be able to access other cores. Such lack of access control may not be desirable as it allows users to perform searches on unauthorized data that may not belong to them.
The Trust No One design philosophy requires that a owner of encrypted data should always remain in control of decrypting the data, and no third party can access the decryption without obtaining authorization from the owner.
To apply the Trust-No-One philosophy in designing a data store, a security model is needed to offer security from 3 different dimensions, namely, data-at-rest trust, superuser trust, and owner trust.
Data-at-rest trust: When data is at rest on a computer readable medium, it is subject to theft and unauthorized physical access to the computer. A conventional solution is to encrypt all data stored in the medium. One drawback of this approach is the need to decrypt all data each time a search request is processed, resulting in performance degradation that only gets worse with increasing data volume and amount of search requests.
Superuser trust: When data is encrypted in the above fashion, it is typically done by means of a superuser key, or the equivalence of a root access. The resulting trustworthiness of data-at-rest hinges on the the system key being trusted fully. In other words, data-at-rest is deemed untrustworthy as soon as the system key is compromised. The first drawback is the violation of trust, as the control of decryption falls into the hands of the superuser, instead of owners of the encrypted data. One alternative solution is to encrypt data by means of owner-keys instead. This approach presents a different drawback in the difficulty to process search requests, which would require decrypting all data by means of obtaining all the respective owner-keys, which is also a violation of trust among users.
Owner trust: yet another trust dimension is to restrict access to encrypted data by ownership. A search request is allowed to be processed only after its associated identity is successfully authenticated to be trustworthy, that the request is trusted to be originated by the data owner or a delegate with equivalent assigned privilege. One drawback is the tight coupling between the identity of an owner and the search request. Using a password authentication as an example, which is a commonly used challenge-response type of technique to authenticate the identity of a user. All it takes is a valid password to process search requests to access encrypted data under the privilege of the owner. One drawback of this approach is the lack of support for sharing access to data without sharing the password, and the sharing of a password results in compromising any trust that may have been associated with an identity.
With the increasing demand for a secure long-term massive data store, it is highly desirable for an invention that can provide a data store with the Trust No One security model, by addressing all of the aforementioned drawbacks.